AIPE for the RMSEA 1 Running head: AIPE FOR THE RMSEA Accuracy in Parameter Estimation for the Root Mean Square Error of Approximation: Sample Size Planning for Narrow Confidence Intervals

The root mean square error of approximation (RMSEA) is one of the most widely reported measures of misfit/fit in applications of structural equation modeling. When the RMSEA is of interest, so too should the accompanying confidence interval. A narrow confidence interval reveals that the plausible parameter values are confined to a relatively small range, at the specified level of confidence. The accuracy in parameter estimation approach to sample size planning is developed for the RMSEA so that the confidence interval for the population RMSEA will have a width whose expectation is sufficiently narrow. Analytic developments are shown to work well with a Monte Carlo simulation study. Freely available computer software is developed so that the methods discussed can be implemented. The methods are demonstrated for a repeated measures design, where the way in which social relationships and initial depression influence coping strategies and later depression are examined. AIPE for the RMSEA 3 Accuracy in Parameter Estimation for the Root Mean Square Error of Approximation: Sample Size Planning for Narrow Confidence Intervals Structural equation modeling (SEM) is widely used in many disciplines, where variables tend to be measured with error and/or latent constructs are hypothesized to exist. The behavioral, educational, and social sciences literature, among others, has seen tremendous growth in the use of SEM in the last decade. The general goal of SEM is to recover the population covariance matrix, Σ, of k manifest (observed) variables by fitting a theoretical model that describes the relationships among the k measured variables and the specified latent variables. The root mean square error of approximation (RMSEA; Steiger & Lind, 1980; Browne & Cudeck, 1992) has become one of the most, if not the most, widely used assessment of misfit/fit in the applications of SEM (e.g., Taylor, 2008; Jackson, Gillaspy, & Purc-Stephenson, 2009). Unlike many other fit indices, the RMSEA is used both descriptively (i.e., sample estimates) and inferentially (with confidence intervals and hypothesis tests). The two most important features of the RMSEA are (a) it is a standardized measure not wedded to the scales of the measured or latent variables; (b) its approximate distributional properties are known, which makes it possible to obtain parametric confidence intervals and perform hypothesis tests. There are two popular ways to use the RMSEA to assess a model fit. The first method is to examine the point estimate and compare it to a certain fixed cutoff value, say c. In this context, if ε̂ < c, the model is considered to have a certain degree of fit (e.g., close fit, mediocre fit, etc.), where ε refers to the population RMSEA and ε̂ refers to the point estimate. The second method is to conduct hypothesis testing to infer if the null hypothesis H0 : ε ≥ c can be rejected at a specified significance level (e.g., MacCallum, Browne, & Sugawara, 1996). If the null hypothesis is rejected, it is concluded that ε < c and the model fit is better than the degree of fit corresponding to the cutoff AIPE for the RMSEA 4 value of the specified null hypothesis. For both of the methods, choosing a proper cutoff value (c) is critically important, and a widely used convention is that ε ≤ 0.05 refers to close fit, ε ≤ 0.08 mediocre fit, and ε > 0.10 poor fit (see, e.g., Browne & Cudeck, 1992; MacCallum et al., 1996). Besides these two conventional methods, a third way to assess the model fit is to form a confidence interval for the population RMSEA. Instead of answering how likely ε > c is, a confidence interval for ε is interested in the value of ε itself, and thus is not wedded to any cutoff value c. In applied research, sample estimates almost certainly differ from their corresponding population parameter. A confidence interval acknowledges such uncertainty and provides a range of plausible values for the population parameter at some specified confidence level (e.g., .90, .95, .99). Confidence intervals “quantify our knowledge, or lack thereof, about a parameter” (Hahn & Meeker, 1991, p. 29), and correspondingly we know more, holding everything else constant, about parameters that have narrow confidence intervals as compared to wider confidence intervals. From a scientific perspective, the accuracy of the estimate is of key importance, and in order to facilitate scientific gains by building cumulative knowledge, researchers should work to avoid “embarrassingly large” confidence intervals (Cohen, 1994, p. 1002) so that the accuracy of the parameter estimate is respectable and appropriate for the intended use. A general approach to sample size planning termed accuracy in parameter estimation (AIPE; e.g., Kelley & Maxwell, 2003; Kelley, Maxwell, & Rausch, 2003; Kelley & Rausch, 2006; Kelley, 2007b, 2007c, 2008) permits researchers to obtain a sufficiently narrow confidence interval so that the parameter estimate will have a high degree of expected accuracy at a specified level of confidence. The AIPE approach to sample size planning is an important alternative or supplement to the traditional power analytic approach (e.g., Cohen, 1988; see Maxwell, Kelley, & Rausch, 2008, for a review and comparison of AIPE and power analysis approaches to sample size planning). In AIPE for the RMSEA 5 structural equation modeling, planning sample size so that the RMSEA is estimated with a sufficiently narrow confidence interval will facilitate model evaluation and descriptions of the extent to which data is consistent with a specified model. In this article we first briefly review confidence interval formation for the population RMSEA and then develop a method to plan sample size so that the expected confidence interval width for the RMSEA is sufficiently narrow. Our sample size planning method is then evaluated with an extensive Monte Carlo simulation study so that its effectiveness is verified in situations commonly encountered in practice. We then show an example of how our methods can be used in an applied setting. Additionally, we have also implemented the sample size planning procedure into R (R Development Core Team, 2010) so that the methods can be readily applied by researchers. Point Estimate and Confidence Interval for the RMSEA In this section we briefly review the confidence interval formation for RMSEA and define our notation. Readers who want to implement the sample size planning methods may wish to only browse this section, as it is not necessary to fully understand the confidence interval formation theories in order to implement the sample size planning methods. Nevertheless, this section is necessary to fully understand the rationale of the methodological developments we make. Let Σ be the population covariance matrix of k manifest variables and S be the sample covariance matrix based on N individuals. Further, let θ∗ be a q × 1 vector of potential parameter values for a postulated covariance structure, where the q values are each identified. The k × k model-implied covariance matrix is denoted M(θ∗). The model’s degrees of freedom, ν, is then k(k + 1)/2 − q. For a correctly specified model, there exists a particular θ∗, denoted θ, such that M(θ) = Σ, AIPE for the RMSEA 6 where θ is the q × 1 vector of the population parameters. Of course, θ is unknown in practice and must be estimated. Estimation of θ can be done in several ways (e.g., maximum likelihood, generalized least squares, asymptotic distribution-free methods), with the most widely used estimation procedure being normal theory maximum likelihood. We use normal theory maximum likelihood estimation throughout the article, and use θ̂ to denote the maximum likelihood estimate of θ. Values for θ̂ can be obtained by minimizing the discrepancy function with respect to θ∗ (e.g., McDonald, 1989; Bollen, 1989) F ( S,M ( θ∗ )) = log (|M (θ∗) |) + tr (SM (θ∗)−1) − log |S| − k, (1) where tr(·) refers to the trace of the matrix and an exponent of −1 denotes the inverse of the matrix. Since θ̂ minimizes Equation 1, min F ( S,M ( θ∗ )) = F ( S,M ( θ̂ )) ≡ F̂, (2) where F̂ is the value of the maximum likelihood discrepancy function evaluated at θ̂. Based on Equation 1, F ( S,M ( θ∗ )) is zero only when S equals M ( θ∗ ) and increases without bound as S and M ( θ∗ ) become more discrepant. For a correctly specified model, when the assumptions of independent observations and multivariate normality hold and sample size is not too small, Steiger, Shapiro, and Browne (1985, Theorem 1; see also Browne & Cudeck, 1992) showed that the quantity T = F̂ × (N − 1) (3) AIPE for the RMSEA 7 approximately follows a central χ2 distribution with ν degrees of freedom. For an incorrectly specified model there exists no θ∗ such that M(θ∗) = Σ. The discrepancy between the population covariance matrix and the population model-implied covariance matrix can be measured as min F ( Σ,M ( θ∗ )) = F (Σ,M (θ0)) ≡ F0, (4) where θ0 is a vector of population model parameters, and F0 is always larger than zero for a misspecified model. For such a misspecificed model, when the assumptions of independent observations and multivariate normality hold, N is not too small, and the discrepancy is not too large, Steiger et al. (1985, Theorem 1; see also Browne & Cudeck, 1992) showed that the quantity T from Equation 3 approximately follows a noncentral χ2 distribution with ν degrees of freedom and noncentrality parameter1 L = F0 × (N − 1). (5) The population RMSEA is defined as